Imitation learning from human feedback studies how to train well-performed imitation agents with an annotator's relative comparison of two demonstrations

In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap.

In this paper, departing from the existing literature on IL for MFGs, we introduce a new solution concept called the Nash imitation gap.

In fact, the interaction of these two aspects requires addressing the fact that each agent's own safety constraint requires information from all others.

In fact, the interaction of these two aspects requires addressing the fact that each agent's own safety constraint requires information from all others.

Q1: ...the claim that the algorithm really manages to align the latent distributions of real and simulated data... We will revise the inappropriate statements in the final version. Q2: In the model adaptation phase, are state-action pairs simply sampled randomly from their respective buffers? Do you have results for a single, monolithic model? Q4: Did you investigate the reasons for the slow learning in the 500 steps on InvertedPendulum compared to PETS? Q1: The experiments shown in Figure 2 do not outperform MBPO beyond the confidence bounds.

Otherwise, we only need to seek the strongest one.

We consider a factored CMDP environment described previously.

Algorithm 1 and implementation details are listed in Appendix E. Algorithm 1 Perfect Alignment for Skew-Fit (P A-SF).

We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kullback-Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-art demonstration efficiency on benchmark control tasks.